Real-time process and technology using image processing to maintain and ensure viewer comfort during capture, live transmission, and post-production of stereoscopic 3D imagery

ABSTRACT

This process and invention may be used for “live” (on-line, recorded, real-time, or transmitted) stereoscopic image generation, post-production, and viewing to ensure the imagery, from the cameras through the systems to the final screens, maintains an acceptable level of viewer comfort and actively causes an action if an image becomes “uncomfortable” to view. The invention will use image-processing hardware and software to monitor stereoscopic 3D image data, and will generate an output response based on qualifying the imagery. The output responses include generating an alarm to alert technical crew members to a “discomfort” condition so they may take corrective action. The alarm may be an audio or visual alert, and warn of the type of discomfort detected. The alarm condition may trigger the automatic control of a video switching device, which would immediately route an appropriate “comfortable” input source to the output. The alarm condition may duplex a single camera or playback signal into an identical stereo pair, which will be 2D, but will be comfortable to view.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to provisional application entitled,REAL-TIME PROCESS AND TECHNOLOGY USING IMAGE PROCESSING TO MAINTAIN ANDINSURE VIEWER COMFORT DURING CAPTURE, LIVE TRANSMISSION, ANDPOST-PRODUCTION OF STEREOSCOPIC 3D IMAGERY, filed Jul. 14, 2005, havinga Ser. No. 60/698,963, which is hereby incorporated by reference in itsentirety.

FIELD OF THE INVENTION

The present invention relates generally to stereoscopic 3D camera andviewing systems.

BACKGROUND OF THE INVENTION

For many years stereoscopic 3D movies have been made. Some have beengood, but many others have been bad. The badly made 3D movies have beenmainly the symptom of inadequate equipment, or the lack of knowledge ofgood stereography. The effectively creates discomfort by the viewer, aswell as having a negative impact on the 3D industry.

In this digital age, it is now feasible to create technologicalsolutions to this dilemma.

This invention describes a process to ensure a comfortable Stereoscopic3D viewing experience, whereby the viewer will observe natural 3Dimagery without the psychovisual effects of eye strain, fatigue, ordiscomfort.

SUMMARY OF THE INVENTION

The process of this invention is performed by extracting usefulinformation from visual content, by means of image-processing the 3Dimagery from a stereoscopic 3D camera system, and provides an outputresponse based on logic decisions and pre-defined thresholds.

The output response includes generating alarms to alert technical crewmembers of a “discomfort” condition so they may take corrective action.These alarms may be an audio or visual alert, and warn of the type ofdiscomfort detected. The alarm condition may also trigger the automaticcontrol of a video switching device, which would immediately route anappropriate “comfortable” input source to the output. The matrixswitcher requires a common banking capability, because stereo pairs arerouted simultaneously to the output.

The alarm condition may duplex a single camera or playback signal intoan identical stereo pair, which will be 2D, but comfortable to view.

(The “comfortable” input source may be a pre-defined or known“comfortable” 3D camera imagery, or other video content from a playbackdevice.)

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a typical signal path of a “live” stereoscopic 3Dinfrastructure, including the processing units involved.

FIG. 2 shows a typical Capture-side detail, in the form of a blockdiagram.

DETAILED DESCRIPTION

One embodiment of this invention (FIG. 1) includes a complete “live”infrastructure, consisting of the following signal path: a stereoscopic3D camera rig/s, an image processing unit to ensure comfortable 3Dcapture, encoding equipment, transmission equipment at the capture side,reception equipment at the viewing side/s, decoding equipment, animage-processing unit to ensure the reception conforms to comfortable 3Dviewing, and a 3D display apparatus such as a 3D projector/s.

During capture, the imagery may be uncomfortable to view due to manyvariables and reasons, and these are quantified by image processingelectronics, which generates alarms based on pre-defined criteria, andalarm thresholds.

When using a remote viewing location, where the video content istransmitted “live” from the 3D camera location, additional imageprocessing electronics is required at the viewing location to ensure theprojectors are fed “comfortable” 3D imagery, in case there is failure ofthe captured quality, during the transmission. The imagery can becorrupted, missing, noisy, or out of sync. This electronics system onthe receiving side would use a subset of the image-processing describedabove, and use a matrix switched output, to guarantee the projectors arefed “comfortable” 3D content.

This process may also be used in post-production (off-line), to ensurerecorded media maintains comfortable 3D imagery, and generates similaroutput responses appropriate for an edit session, and is especiallyuseful for finding transition points that are “comfortable” forclose-matching 3D.

This invention uses a mathematical model to quantify acceptable rangesof comfort.

The capture-side (FIG. 2) input into to the system may include:

-   1) Imagery from the left camera of the stereoscopic 3D camera rig,    to be image processed.-   2) Imagery from the right camera of the stereoscopic 3D camera rig,    to be image processed.-   3) Focus metadata from the 3D rig.-   4) Iris metadata from the 3D rig.-   5) Zoom metadata from the 3D rig.-   6) Inter-Ocular metadata from the 3D rig.-   7) Convergence metadata from the 3D rig.-   8) Screen dimensions. (for TV, theater, IMAX, etc)-   9) Distance range between this screen and the viewers.-   10) Acceptable image horizontal disparity, which may be expressed as    a percentage of total image size.-   11) Fusional range.-   12) Alarm thresholds:-   a) Gross difference (non-“fusable”)-   b) Focus disparity-   c) Luminance disparity-   d) Chrominance disparity-   e) Magnification disparity-   f) Telecentricity disparity-   g) “Broken frame” acceptance level-   h) Vertical content weighting factor-   i) Vertical disparity (expressed as number of lines, angle, or    Percentage of screen height)

To generate an alarm, the system's image-processing function will firstlook for obvious image errors such as missing video from a camera orcameras, or out-of-sync video, either sub-frame or multiple frame.

Then obvious lens mismatch is processed. Focus disparity is calculated,where the image-processing algorithm includes edge detection and/or ahigh-pass filtering to narrow in on the highest frequency detail of thechart. Luminance disparity (created by iris, gamma, black-level, knee,etc.) is calculated, where the image-processing algorithm includes imagesubtraction and/or image correlation. Chrominance disparity (hue,saturation) is calculated, where the image-processing algorithm includescolor matrix conversion, and image correlation. Alarms are generated ifthe mismatches exceed pre-defined thresholds.

Then by using disparity mapping, by block and pixel methods, a depth mapis created. This is done using a standard neural-net process, wherestrong links are magnified (for adjacency) by parallel vector analysisto find stereoscopically “fusable” subject matter. Broken links in theneural-net will be determined to be caused by either “breaking frame” onthe boundary of the images, or from stereoscopic occlusion within theimages. The “breaking-frame” condition has an adjustable acceptancelevel, or alarm threshold.

“Blob analysis” algorithms are used to combine any linked “fusable”subject matter into bigger “blobs”.

The amount of “fusable” subject matter, as an area ratio of the fullscreen size, is used to determine if there is a gross difference fromboth camera views, which may be caused by something obstructing the viewof one camera. If this gross difference is sufficient to exceed thealarm threshold, an alarm condition will be generated.

The “blobs are analyzed for and magnification disparity (zoom mismatch),and telecentricity mismatch, upon which an alarm will be generated ifthese mismatches exceed the alarm thresholds.

The range of all angles to the boundaries of “fusable” subject matter,or “fusional range” are calculated, and if any angle exceeds the alarmthreshold, an alarm will be generated. These angles to the “fusable”subject matter are performed in the horizontal plain, as this is thenatural stereoscopic horizontal disparity. Excessive horizontaldisparity, either towards viewer divergence or excessive convergence,will generate an alarm.

The search range of the neural net will include several lines above andbelow the present search range, to extract possible vertical orrotational disparity, upon which an alarm will be generated if thevertical disparity is found to exceed the alarm threshold, and takesinto account the screen size.

The background will be searched for a concentration of vertical content(such as lamp posts, or a fence line). A Fourier transform is performedin the horizontal direction to extract this in the frequency domain.This area of the image will be considered less stereoscopically“fusable”, and weighted accordingly, taking into account other “fusable”subject matter. An alarm will be generated if it exceeds a pre-definedthreshold.

Finally, the remaining uncategorized areas will be deemed occlusionareas, and will be ignored, because they are naturally stereoscopic.

The alarm condition may also trigger the automatic control of a videoswitching device, which would immediately route an appropriate“comfortable” input source to the output.

1. A process of using image processing to qualify stereoscopic 3Dimagery, with respect to its psychovisual comfort level.
 2. A method ofclaim 1, where the process is performed in hardware in real-time.
 3. Amethod of claim 1, where the process is performed in software inreal-time.
 4. A method of claim 1, where the response output of thesystem is an alarm.
 5. A method of claim 4, where the alarm is anaudible signal.
 6. A method of claim 4, where the alarm is an visualsignal.
 7. A method of claim 4, where alarms are generated based onimage criteria, derived from 3D stereoscopic imagery.
 8. A method ofclaim 4, where alarms are generated based on metadata values supplied bythe 3D rig motion control system.
 9. A method of claim 4, where alarmsare generated based on pre-defined thresholds of image attributes.
 10. Amethod of claim 4, where alarms are used to control the switching of avideo switcher, to switch to a stereographically comfortable videosource.
 11. A method of claim 4, where the alarm is based on predefinedinput constants.
 12. A method of claim 11, where a predefined constantis the viewing screen size.
 13. A method of claim 11, where a predefinedconstant is the viewing distance, or range of distances to the screen.14. A method of claim 7 and/or claim 8, where the image criteria is thefocus function of the imagery.
 15. A method of claim 7 and/or claim 8,where the image criteria is the iris function of the imagery.
 16. Amethod of claim 7 and/or claim 8, where the image criteria is the zoomfunction of the imagery.
 17. A method of claim 7 and/or claim 8, wherethe image criteria is the inter-ocular function of the imagery.
 18. Amethod of claim 7 and/or claim 8, where the image criteria is theconvergence function of the imagery.
 19. A method of claim 9, where thealarm threshold sets a limit to the acceptable horizontal disparityoffset between the imagery from each camera.
 20. A method of claim 9,where the alarm threshold sets a limit to the acceptable verticaldisparity offset between the imagery from each camera.
 21. A method ofclaim 9, where the alarm threshold sets a limit to the acceptablestereographic fusional range.
 22. A method of claim 9, where the alarmthreshold sets a limit to the acceptable gross difference(non-“fusable”)between both cameras.
 23. A method of claim 9, where thealarm threshold sets a limit to the acceptable stereographic focusdisparity.
 24. A method of claim 9, where the alarm threshold sets alimit to the acceptable stereographic luminance (iris) disparity.
 25. Amethod of claim 9, where the alarm threshold sets a limit to theacceptable stereographic chrominance disparity.
 26. A method of claim 9,where the alarm threshold sets a limit to the acceptable stereographicmagnification (zoom) disparity
 27. A method of claim 9, where the alarmthreshold sets a limit to the acceptable stereographic telecentricitydisparity.
 28. A method of claim 9, where the alarm threshold sets alimit to the acceptable of a stereographic “Broken frame” at the edgesof each frame.
 29. A method of claim 9, where the alarm threshold sets alimit to the acceptable amount of “Vertical content”, such as fenceposts, in the imagery.
 30. A method of claim 10, where the comfortablesterepscopic content is a 2D view, by making the left-eye and right-eyeviews identical imagery.